Skip to content

Change from TextFileReader to ParquetStreamReader#348

Open
JanWillruth wants to merge 49 commits intoglamod:mainfrom
JanWillruth:reader_io
Open

Change from TextFileReader to ParquetStreamReader#348
JanWillruth wants to merge 49 commits intoglamod:mainfrom
JanWillruth:reader_io

Conversation

@JanWillruth
Copy link
Collaborator

@JanWillruth JanWillruth commented Jan 15, 2026 •
edited by ludwiglierhammer
Loading

To do

  • Change from TextFileReader to ParquetStreamReader for (better) handling of files larger than RAM when chunk_size is specified
  • Rework affected Databundle code
  • Move ParquetStreamReader from mdf_reader.utilis.utilities to common.iterators
  • Rework affected cdm_mapper.mapper code
  • Rework affected common.select code
  • Rework affected common.inspect code
  • Rework affected metmetpy.validate code
  • Rework affected metmetpy.correct code
  • Rework affected mdf_reader.utilities.utils code
  • Rework affected mdf_reader.writer code
  • Rework affected core._utilities._copy code
  • Add ParquetStreamReader option to common.replace code
  • Remove common.pandas_TextParser_hdlr
  • clean up / update testing suite -> see re-work testing suite #365
  • do not use 'make_parser' in testing suite -> see re-work testing suite #365
  • Rework docstrings -> see enforce mypy hook #368
  • Rework type hints -> see enforce mypy hook #368

Ideas

  • write a decorator so that we can call a function in the "normal" way, but the decorator decides whether to execute the function or pass it to common.iterators.process_disk_backed

    • see _apply_or_chunk but as a decorator
    • e.g. both is valid my_func(DataFrame, *args, **kwargs) and my_func(ParquetStreamReader, *args, **kwargs)

Issues

This PR addresses opened issues:

@JanWillruth
Change from TextFileReader to ParquetStreamReader for (better) handli…
c09a06f 
…ng of files larger than RAM when chunk_size is specified; Rework affected Databundle code
@github-actions
Copy link

github-actions bot commented Jan 15, 2026

Warning
This Pull Request is coming from a fork and must be manually tagged approved
in order to perform additional testing.

@JanWillruth
Save columns schemas alongside parquet to restore MultiIndex column n…
c266a4d 
…ames; Remove unneeded TextFileReader tests form test_pandas.py
@ludwiglierhammer
Copy link
Collaborator

ludwiglierhammer commented Jan 28, 2026

@JanWillruth: I made some high performance tests. This PR does not really affect the maximum memory usage, but speeds up the code a liitle bit. Nevertheless, we should use this PR your readability reasons.

Do you want to move the ParquetStreamReader to common. Then, we could have the ParquetStreamReader at one general place and could make use of it in common.select, mdf_reader and cdm_mapper.

@ludwiglierhammer ludwiglierhammer mentioned this pull request Jan 28, 2026
Merged
2 tasks
@ludwiglierhammer
Copy link
Collaborator

ludwiglierhammer commented Jan 29, 2026

I merged #360 into the main branch. Please resolve the merge conflicts and we can focus on this PR again.

JanWillruth and others added 3 commits January 29, 2026 11:57
@JanWillruth
Merge remote-tracking branch 'origin/main' into reader_io
8588068 
# Conflicts:
#	cdm_reader_mapper/mdf_reader/utils/utilities.py
#	tests/test_reader_utilities.py
@ludwiglierhammer
try to use ParquetStremReader
a6e7b86
@ludwiglierhammer
re-add make_copy for TextFileReader objects
858b4ca
@ludwiglierhammer ludwiglierhammer mentioned this pull request Jan 30, 2026
Open
@ludwiglierhammer
Copy link
Collaborator

ludwiglierhammer commented Jan 30, 2026

Hi @jtsiddons, what do you think about this PR? Some ideas for improvements or generel comments?

@jtsiddons
Copy link
Collaborator

jtsiddons commented Jan 30, 2026

Hi @jtsiddons, what do you think about this PR? Some ideas for improvements or generel comments?

Thanks @ludwiglierhammer - I've scheduled some time to have a look this afternoon

@ludwiglierhammer ludwiglierhammer mentioned this pull request Jan 30, 2026
Closed
ludwiglierhammer
ludwiglierhammer reviewed Jan 30, 2026
View reviewed changes
This was referenced Jan 30, 2026
Merged
Open
@ludwiglierhammer ludwiglierhammer linked an issue Feb 10, 2026 that may be closed by this pull request
Refactor logic to handle chunking outside of reader(/mapper) for readability/maintenance #349
Open
@ludwiglierhammer
Copy link
Collaborator

ludwiglierhammer commented Feb 13, 2026 •
edited
Loading

Hi @jtsiddons, we could replace all TextFileReader elements with the new ParquetStreamReader. Do you have any further suggestions for this PR. We would appreciate your review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ludwiglierhammer ludwiglierhammer ludwiglierhammer left review comments

Assignees

No one assigned

Labels

cdm_mapper common docs information mdf_reader metmetpy

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Refactor logic to handle chunking outside of reader(/mapper) for readability/maintenance TextFileReader needed?

3 participants

@JanWillruth @ludwiglierhammer @jtsiddons

Comments